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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE [31 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- tf the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 
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- Failure to reply within the set or extended period for reply by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 
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1 )M Responsive to communication(s) filed on April 15. 2005 . 
2s)M This action is FINAL. 2b)n This action is non-final. 

3) 0 Since this application is in condition for allowance except for fomial matters, prosecution as to the merits is 

closed in accordance with the practice under £x parte Quay/e, 1935 CD. 11, 453 O.G. 213. 
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Replacement drawing sheet(s) including the correction is required If the drawing(s) is objected to. See 37 CFR 1.121(d). 
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UNITED States Patent AND Trademark Office 

P.O. Box 1450, Alexandria, Virginia 22313-1450 - www.uspto.gov 

Examiner's Detailed Office Action 
Response to Amendment 

1. This Office Action is responsive to communication received on April 15, 2005. 
Amendment under 37 CFR § 1.111. Reconsideration and allowance of the present application 
09/787,698, is respectfully requested by applicant. All such supporting documentation has been 
placed in applicant's file. 

2. Applicant's remarks have been folly considered, however, they are not persuasive. 

3. Claims 22 & 36 stand rejected under Title 35 USC § 102(b) & claims 23-31, 33-38, and 
40-42, stand rejected under Title 35 USC § 103(a), the con5)lete text has been included below. 

Claim Rejections - 35 USC § 102 

4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed publication in this 
or a foreign country, before the invention thereof by the applicant for a patent. 
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5. Claim 22 are rejected under 35 U.S.C. 102(b) as being anticipated by 

Ralph Neuneier, "Enhanced Q-Leaming for Optimal Asset Allocation" Advances in Neural 
Information Processing Systems (1997). 

Regarding Claim 22. Neuneier teaches, A method for computer-aided determination of a 
sequence of actions for a system having states, the method comprising the steps of: performing a 
transition in state between two states on the basis of an action [(2.1 Q-Leaming for asset 
alloca- tion, page 937-938)]; determining the sequence of actions to be performed such that a 
sequence of states results from the sequence of actions [(2.1 Q-Leaming for asset allocation, 
page 937-938)]; optimizing the sequence of steps with re- gard to a prescribed optimization 
function, including a variable parameter [(2.1 Q-Leaming for asset allocation, page 937-938)]; 
and using the variable para- meter to set a risk which the resuhing sequence of states has with 
respect to a prescribed state of the system. [(3.1 Risk-adjusted MDP's, page 941)] 

Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 
102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the 
subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill 
in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

7. Claims 23-3 1 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ralph Neuneier, "Enhanced Q-Learaing for Optimal Asset Allocation" Advances in Neural 
Information Processing Systems (1997) further in view of Werbos (USPN 6,169,981). 
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The Neuneier reference has been discussed above and does not explicitly teach the limitation 
of claims 23-24, 26, 29-30. However, Werbos teach the limitations of claims 23-24, 26, 29-30. 

Regarding Claim 23. Werbos teaches, using approximate dynamic programming for the purpose 
of determination. [(coL 7, line 37-38 "In Approximate Dynamic Programming (ADP) ... model 
to approximate this function J.")] It would have been obvious at the time the invention was made 
to a person having ordinary skill in the art to which said subject matters pertains, to employ 
approximate dynamic programming (ADP), because, ADP allows for general-purpose 
programming for doing optimization over time by using learning and approximation. Moreover, 
ADP has emerging as one of the most promising mathematical and conputational approaches to 
resolve the problem of nonhnear, large scale dynamic control problems under uncertainty. 

Regarding Claim 24. Werbos teaches, basing the approximate dynamic programming Q-leaming. 
[(coL 93, line 38-50 "The basic equation for "P' given in ...P. Werbos, ... Still, these designs are 
all a step up from the 1983 BSA design,")] It would have been obvious at the time the invention 
was made to a person having ordinary skill in the art to which said subject matters pertains, to 
errqDloy approximate dynamic programming (ADP) Q-Leaming, because, Q-Leaming is a 
stochastic approximation-like method that iterates on the Q-factors. 

Regarding Claim 25. Neuneier teaches, forming an optimization function within Q-leaming in 
accordance with the following rule: . . . [(as per applicant's own admission (page 2-4 )] It 
would have been obvious at the time the invention was made to a person having ordinary skill in 
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the art to which said subject matters pertains, to employ an optimization function for specifying 
the expectation and gains accumulated over time. 

Regarding Claim 26. Werbos teaches, basing the approximate dynamic programming on TD(8)- 
leaming. [(coL 110, line 39-45 "/w any event, the procedure implied ... is considerably more 
general than any of the original formulation.'')] It would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matters 
pertains, to employ approximate dynamic programming on TD(8)-leaming, because, the 
temporal difference method is a type of reinforcement leaming that learn through exploiting the 
difference in evaluating actions in successive steps and thus handling sequences in an 
incremental manner. 

Regarding Claim 27. Neuneier teaches, forming an optimization function within Q-leaming in 
accordance with the following rule: ... [(as per applicant's own admission (page 8-10 )] It 
would have been obvious at the time the invention was made to a person having ordinary skill in 
the art to which said subject matters pertains, to employ an optimization function for specifying 
the expectation and gains accumulated over time. 



Regarding Claim 28. Neuneier teaches, using a technical system to determine the sequence of 
actions before the determination measured values are measured, [(as per applicant's own 
admission (page 1)] It would have been obvious at the time the invention was made to a person 
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having ordinary skill in the art to which said subject matters pertains, to predict a planning 
mechanism to determine the sequence of actions to reach the chosen goal state. 

Regarding Claim 29. Werbos teaches, subjecting the technical system to open-loop control in 
accordance with the sequence of actions. [(coL 100, line 20-28 "In summary, the concept of 
learning—while not all-encompassing—could be expected to give us a parsimonious 
understanding ... such research is often useful only when it is integrated into a specific plan to 
use the sensors within a larger control loop.")] It would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matters 
pertains, to employ open-loop control, because an open-loop control system is controlled 
directly, and only, by an input signal, without the benefit of feedback. The basic units of this 
system are an ampUfier and a motor. The amplifier receives a low-level input signal and 
amplifies it enough to drive the motor to perform the desired job. Open- loop control systems are 
not as commonly used as closed-loop control systems because they are less accurate. 

Regarding Claim 30. Werbos teaches, subjecting the technical system to closed-loop control in 
accordance with the sequence of actions. [(coL 130, line 19-27 "Balakrishnan has mainly studied 
problems in aircraft and missile control ... done tests demonstrating robustness and closed-loop 
"re-adaptation")] It would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matters pertains, to enploy closed-loop 
control, because process improvement can be obtained by means of closed-loop control i.e., with 
reliable. While, the basic idea of feed-back control is to correct disturbances, which may appear 
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in real production processes. With reliable closed-loop control the process can often be run at 
paths in the state space which promise a better process performance. 

Regarding Claim 3 1 . Neuneier teaches, modeling the system as a Markov decision problem, [(as 
per applicant's own admission (page 1)] It would have been obvious at the time the invention 
was made to a person having ordinary skill in the art to which said subject matters pertains, to 
employ a natural approach to developing control architectures for complex tasks is to design 
multiple simple components, each addressing one aspect of the task domain, and provide a 
framework for combining the results. When an agent must leam these components by interacting 
with the environment, as in a Markov decision problem, the designer must take care not to 
sacrifice an optimal solution for a convenient representation. 

8. Claims 33-35 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ralph Neuneier, "Enhanced Q-Learning for Optimal Asset Allocation" Advances in Neural 
Information Processing Systems (1997), further in view oi Howard (USPN 6,336,109). 

Regarding Claim 33. Howard teaches, using the system as a communications system. [(coL 7, 
line 65 to col. 8, line 5 "Another possibility is to use a neural network classifier ...specifically 
for telecommunications applications.")] It would have been obvious at the time the invention 
was made to a person having ordinary skill in the art to which said subject matters pertains, to 
employ a processor is used in a communications system, because the processor {computer 
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Pages 



system) can be used to analyze data about the transmission of messages in a communications 
system 

Regarding Claim 34. Howard teaches, using the system to carry out access control in a 
communications network. [(coL 1, line 11-16 "In the telecommunications field, large amounts of 
data are available, for example about customer behavior and telephone usage. This data 
contains potentially useful information for many purposes such as detection of fraud, marketing, 
billing, maintenance planning and fault detection.")] It would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matters 
pertains, to employ a processor is used to carry out access control in a communications network, 
because the processor {computer system) can be used to interpret, analyze and investigate user 
patterns of behavior in detecting e.g., telecommunications fraud. 

Regarding Claim 35. Howard teaches, using the system to carry out a routing in a 
communications network. [(coL 1, line 11-16 "In the telecommunications field, large amounts of 
data are available, for example about customer behavior and telephone usage. This data 
contains potentially useful information for many purposes such as detection of fraud, marketing, 
billing, maintenance planning and fault detection")] It would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matters 
pertains, to en5)loy a processor is used to carry out routing in a communications network, 
because the processor {computer system) can be used to interpret, analyze and investigate user 



Application/Control Number: 09/787,698 Page 9 

Art Unit: 2121 

patterns of behavior in detecting e.g., in relation to faults in a communications network and 
encryption key management. 



9. Claim 36 is rejected under 35 U.S.C. 102(b) as being anticipated by 

Walter J. Gutjahr, "Failure Risk Estimation via Markov Software Usage Models" Department of 

Statistics, Operations research and Computer Science, University of Vienna, Austria (1997). 

Regarding Claim 36. Gutjahr teaches, A system for determining a sequence of actions for a 

m 

system having states, wherein a transition in state between two states is performed on the basis of 
an action, the system conqjrising: a processor for determining a sequence of actions, whereby a 
sequence of states resulting from the sequence of actions is optimized with regard to a prescribed 
optimization function, and the optimization function includes a variable parameter for setting a 
risk which the resulting sequence of states has with respect to a prescribed state of the system. 
[(Abstract "... risk estimate ...) and (5. Experimental results, page 189-190, Table 1)] 



Claim Rejections - 35 USC § 103 

10. The following is a quotation of 35 U.S.C. 103(a) >vhich forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 
102 of this title, if the differences between the subject matter sou^t to be patented and the prior art are such that the 
subject matter as a whole would have been obvious at the time the] invention was made to a person having ordinary skill 
in the art to which said subject matter pertains. Patentability shalltnot be negatived by the manner in which the 
invention was made. 
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1 1 . Claims 37-38 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Walter J, Gutjahr, "Failure Risk Estimation via Markov Software Usage Models" Department 
of Statistics, Operations research and Conq^uter Science, University of Vienna, Austria (1997), 
further in view of Werbos (USPN 6,169,981). 

The Gutjahr reference has been discussed above and does not exphcitly teach the limitation 
of claim 37-38. However, Werbos teaches the limitations of claims 37-38. 

Regarding Claim 37. Werbos teaches, the processor is used to subject a technical system to open- 
loop control. [(coL 100, line 20-28 ''In summary, the concept of learning—while not all- 
encompassing—could be expected to give us a parsimonious understanding ... such research is 
often useful only when it is integrated into a specific plan to use the sensors within a larger 
control /oop.")] It would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matters pertains, to en:q)loy open-loop 
control, because an open-loop control system is controlled directly, and only, by an input signal, 
without the benefit of feedback. The basic units of this system are an amplifier and a motor. The 
amplifier receives a low-level input signal and amplifies it enough to drive the motor to perform 
the desired job. Open-loop control systems are not as commonly used as closed-loop control 
systems because they are less accurate. 
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Regarding Claim 38. Werbos teaches, the processor is used to subject a technical system to 
closed-loop control. [(coL 130, line 19-27 "Balakrishnan has mainly studied problems in aircraft 
and missile control ... done tests demonstrating robustness and closed-loop "re-adaptation,'')] It 
would have been obvious at the time the invention was made to a person having ordinary skill in 
the art to which said subject matters pertains, to employ closed-loop control, because process 
improvement can be obtained by means of closed-loop control i.e., with reUable. While, the basic 
idea of feed-back control is to correct disturbances, which may appear in real production 
processes. With reliable closed-loop control the process can often be run at paths in the state 
space which promise a better process performance. 



12. Claims 40-42 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Walter J, Gutjahr, "Failure Risk Estimation via Markov Software Usage Models" Department 
of Statistics, Operations research and Computer Science, University of Vienna, Austria (1997), 
fixrther view of Howard (USPN 6,336,109), 

Regarding Claim 40. Howard teaches, the processor is used in a communications system. [(coL 
7, line 65 to col. 8, line 5 "Another possibility is to use a neural network classifier ...specifically 
for telecommunications applications ")] It would have been obvious at the time the invention 
was made to a person having ordinary skill in the art to which said subject matters pertains, to 
employ a processor is used in a communications system, because the processor (computer 
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system) can be used to analyze data about the transmission of messages in a communications 
system. 

Regarding Claim 41. Howard teaches, the processor is used to carry out access control in a 
communications network. [(coL 1, line 11-16 "In the telecommunications field, large amounts of 

1 

data are available, for example about customer behavioriand telephone usage. This data 
contains potentially useful information for many purposes such as detection of fraud, marketing, 
billing, maintenance planning and fault detection.")\lX would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matters 
pertains, to en^loy a processor is used to carry out access control in a communications network, 
because the processor {computer system) can be used to interpret, analyze and investigate user 
patterns of behavior in detecting e.g., telecommunications fraud. 

Regarding Claim 42. Howard teaches, the processor is used to carry out routing in a 
communications network. [(coL 1, line 11-16 "In the telecommunications field, large amounts of 
data are available, for example about customer behavior and telephone usage. This data 
contains potentially useful information for many purposes such as detection of fraud, marketing, 
billing, maintenance planning and fault detection")] It would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matters 
pertains, to employ a processor is used to carry out routing in a communications network, 
because the processor {computer system) can be used to interpret, analyze and investigate user 
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patterns of behavior in detecting e.g., in relation to faults in a communications network and 
encryption key management. 

Response to Remarks 

13. Examiner interprets the indication of risk set forth by Neuneier (page 941) sufficient to 
indicate that it is known. Moreover, examiner interprets Gutjahr use of statistical analysis suffi- 
ciently employing the set of possible values for a random variable together with a probability 
measure defining the likelihood of those values. 

Examiners Summary 

14. Accordingly, TfflS ACTION IS MADE FINAL. See MPEP § 706.07(a). AppHcant is 
reminded of the extension of time poUcy as set forth in 37 CFR 1 .136(a). 

15. A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 

CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 
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Correspondence Information 



16. Any inquires concerning this communication or earlier communications from the 
examiner should be directed to Michael B. Holmes, who may be reached Monday through 
Friday, between 8:00 a.ni and 5:00 pm EST. or via telephone at (571) 272-3686 or facsimile 
transmission (571) 273-3686 or email Michael holmesb@uspto. go v . 

If you need to send an Official facsimile transmission, please send it to (703) 746-7239. 

If attempts to reach the examiner are unsuccessful the Examiner's Supervisor, Anthony 
Knight, may be reached at (571) 272-3687. 

Hand-delivered responses should be delivered to the Receptionist @ (Customer Service 
Window Randolph Building 401 Dulany Street Alexandria, VA 22313), located on the first floor 
of the south side of the Randolph Building. 
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